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Results of the study show that the answer to question (1) above was 
"yes” ; in answer to question (2) , nine questions in the pretest 
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INTRODUCTION AND PURPOSE 



The study on which this report is based originated with the 
request of the College Board Committee of Examiners in Mathematics 
that the usual statistical analysis of new test questions be supple- 
mented by an attempt to find out what a student actually thinks as 
he solves the questions. The committee suggested that the conjec- 
tures about thought processes that are stimulated by item analysis 
results should be checked occasionally, if not regularly, against 
information obtained by the in-depth interviewing of students. At 
its April 1970 meeting, the committee recommended that a small-scale 
study be conducted to evaluate the merit of this approach. Arrange- 
ments were therefore made for members of Educational Testing Service 
to interview students in May 1970. 

This study permitted the comparison of two theoretically 
different kinds of information about performance on test questions — 
the clinical and the normative. Standard item analysis provides 
information about the average performance of typical groups, but not 
about the procedure by which the examinee arrives at an answer or 
about the specific characteristics of a question that result in its 
being easy or difficult. In contrast to the normative information 
based on the statistical analysis of group data is the clinical 
information derived from the in-depth interviewing of individual 
students. The study reported here attempted to assess the usefulness 
of supplementing the normative information provided by item analysis 
with the clinical information derived from in-depth interviewing. 

The authors of this report found no studies that used the interview 
technique for this purpose, although they did not make an exhaustive 
search of the literature. Three studies (Bloom and Broder, 1950; 
Connolly and Wantman , 1964; and Gentile, 1966) made use of the 
" think-aloud" approach; however, none of these studies had as its 
purpose the revision of individual test questions. 

The purposes of the study are summarized by the following 
questions : 

1. Does the interview technique provide information that 
cannot be obtained from the usual normative approach? 

2. Does the interview technique provide information leading 
to the revision of mathematics test questions? 



Are the revised questions better than the original ver- 
sions in any one of the following ways? 

a. The revised version has appreciably better statistical 
characteristics than the original version when both 
are subjected to a standard normative item analysis, 

b. The students, when interviewed, say that the revised 
version is clearer than the original version. 

c. The interviewers result in the correction of an 
error in the question. 

3. Does the interview technique provide information about 
the extent to which the student has been exposed in a 
mathematics course to the topic, concept, or skill that is 
central to the correct solution of a particular question? 



METHOD 



The College Board Mathematics Level II pretest used in this 
study consists of 25 questions to be administered in 40 minutes. 

This pretest is not typical of most pretests designed for the Level 
II population in that it contains more questions that test sophis- 
ticated concepts or terminology. Approximately one-third of the 
questions were specifically included by the committee to obtain 
information concerning the understanding of these concepts and the 
knowledge of technical terminology. 

Two structured documents for recording information during the 
interviews were prepared: an Item Interview Record and a Test 

Interview Record. A copy of each of the forms is in Appendix A. 

For the purpose of systematically organizing and recording 
student interview information, a notebook consisting of 25 Item 
Interview Records (one for each item in the pretest) and one Test 
Interview Record was prepared for each of the 15 interviews. In 
addition, each interview was tape recorded. 

Two alternative procedures for collecting data were consid- 
ered: (1) asking students to think aloud as they attempted to 

solve the problems in the pretest and (2) interviewing selected 
students after a group administration of the test under regular con- 
ditions. It was decided to follow the latter alternative because 
this more nearly replicated actual test-taking conditions. (See 
Appendix B for details of procedures used in carrying out the study.) 
Before the test was administered, the students were told the purpose 
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of the study without reference to the specific testing program. They 
were also told that if they were interviewed, they would be asked to 
recall their methods of solution and would be given their test book 
so that they could refer to the questions and any notes they had 
written in the book. 

When it was pointed out that this study provided students with 
a chance for involvement in testing processes that affect them, they 
seemed to be motivated to perform well on the test. In fact most of 
the students seemed eager to be interviewed. 

* 

Table 1 describes the schools and the students involved in the 
study. More detailed statistical summaries may be found in Appendix 
C. 



It can be seen from these data that the sample of schools 
ranged from quite strong to relatively weak. Within each school, 
the interview sample was chosen on the basis of test scores and 
information provided by the teacher. At each school an attempt was 
made to include in the interview sample at least one student who 
scored above the average student tested at that school, one who 
scored about average, and one who scored below average. It was soon 
discovered, however, that not much information about the methods of 
solution was obtainable from the students scoring below 10, so more 
able students were used as the study progressed. The distribution 
of scores in the interview sample ranged from 5 to 24. The distri- 
bution of scores appears in Appendix C. The interview sample 

included both boys and girls and minority-group representation. 
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S'UMMARY AND CONCLUSIONS 



The results of this study can be summarized in part by recon- 
sidering the questions that were posed earlier in this report. 

Ques t ion 1 , Does the interview technique provide information that 
cannot be obtained from the usual normative approach? 

From the evidence gathered in the 15 interviews, the answer 
to this question is "yes . 11 For each of the questions, the inter- 
viewers gained some insight into the thought processes of the stu- 
dents as they solved the problems. In addition, the following gen- 
eral statements can be made as a result of interviewees f comments 
regarding individual questions, 

a. Qualifiers or restrictive phrases such as xy ^ 0 that are 
necessary to solve the problem were in general ignored by 
the interviewees. As a rule they realized that the qual- 
ifiers were included in the wording of the questions for 

a purpose but did not always understand the purpose. 

They usually attempted to select the "best answer" in some 
sense completely disregarding the qualifiers. Although 
these qualifiers are ignored, it may still be necessary to 
retain them in some of the questions in order to make the 
questions mathematically sound. 

b. Students at this level do not in general understand so- 
phisticated mathematical phraseology such as "all but 
finitely many" and "if and only if." Because of the lack 
of mathematical sophistication of students at this level, 
perhaps such terminology should be avoided whenever pos- 
sible . 

c. Frequently students did not solve a problem using the 
method tiv£ item writer intended, especially if the intend- 
ed method was an "ins i ghtf ul 11 one. Generally they 
attempted to apply a routine technique or procedure to a 
given problem rather than to think of a quick or insight- 
ful way of solving it. Unless questions can be devised 
that can only be solved by nonroutine methods, so-called 
insightfulness will probably not be a contributing 
factor in the scores of most students. 

d. There is a tendency on the part of some students to sub- 
stitute options whenever possible; however* many students 
resorted to this method only if they were unable to think 
of a direct approach to the problem. It is the opinion 

of the interviewers that the method of substitution is not 



necessarily inappropriate since it indicates that the 
student has some understanding of the concepts involved; 
however, the specifications for a test probably should 
indicate the maximum allowable number of questions that 
can be solved by substitution. 

Question 2 . Does the interview technique provide information lead- 
ing to the revision of mathematics test questions? 

As a result o£ the interviews, 9 questions in the pretest 
were revised (items 2, 10, 1 4 , 18 , 20, 2 1 , 22 , 23, and 24) . The 

original and revised versions of the items can be found in Appendix 
D. Only 4 of these revisions (items 10, 14, 18, and 23) resulted in 

substantial improvements in the statistical characteristics of the 
item for the pretest population. The lack of conclusive "statistical 
improvement,” however, may just be an indication that the normative 
procedure is not as sensitive to difficulties that students have 
with nuances of wording as is the interview technique. 

Q ues t ion 3 . Does the interview technique provide information about 
the extent to which the student has been exposed in a 
mathematics course to the topic, concept, or skill that 
is central to the correction solution of a particular 
prob lem? 

The answer to this question is a definite "yes,” although it 
raises the statistical question of gene rali zabil i ty to the candidate 
population from a limited sample. Certainly powerful clues are 
obtained through the interview process which might be corroborated 
by means of a candidate survey, or to some extent by an item analy- 
sis . 



In addition, the following general statements can be made 
regarding the interviewees 1 reaction to the test as a whole. 

1. The interviewees in general seemed to think that the test 
was at an appropriate level of difficulty, although a few 
of the students who had extremely low scores on the test 
considered it very difficult. 

2. Although most interviewees found some topics tested that 
were unfamiliar to them, they stated that the coverage of 
the test was for the most part appropriate for them. 

3. Most of the interviewees stated that they had sufficient 
time; however, most of them also indicated that they had 
skipped questions that they were unsure of or that they 
thought were too time-consuming. 

4. Most interviewees indicated that they thought the test 
was fair, although a few students considered one or two of 



io 



the questions '‘tricky" or unreasonable. None of the 
students were concerned about the fact that they were 
unable to cope with some of the questions. They seemed 
to accept the idea that “not all students are expected to 
answer all of the questions." 

5. Students general! y objected to questions with a format 

like that of question 18. They felt that they r.'ight have 
knowledge about the concepts involved in the question and 
be able to ascertain whether or not one or more of the 
statements is true or false and still not obtain the 
correct response to the question. 

SUGGESTIONS FOR FUTURE RESEARCH 



It is the opinion of the interviewers that the information 
obtained from this study could be obtained by interviewing fewer 
students provided the students are selected from a variety of 
schools. For this particular study, in which a major purpose was to 
obtain information about the cognitive processes used in solving 
mathematics questions, just as much could have been learned from 
interviewing one average and one above-average student from each 
school. Very little was learned from the poor students; however, 
this might not be the case if the interviewing were being done for a 
different Program or for a different purpose. 

It is also the opinion of the interviewers that the item 
interview and test interview forms (Appendix A) used in this study 
should be revised. The item interview form was too complex; it was 
impossible for one person to complete the entire form during the 
interview. If the study were to be redone, number 3 would probably 
be deleted from the form. An attempt would also be made to structure 
the interview more by devising specific questions about each of the 
items to ask all students being interviewed. Many of the questions 
on the test interview form asked for information that duplicated that 
already obtained in the item-by-item interview. This form would 
probably be reduced to the questions contained in Section II. 

Because of the amount of ETS staff time that would be required 
to conduct interviews on the large quantities of items that are pre- 
tested each year, it would not be practical to use the interview 
procedure for all items. The interviewers are of the opinion, how- 
ever, that conducting this kind of study periodically is justified 
because of the kinds of information that can be obtained about the 
questions and the mathematics curricula in the schools. The inter- 
viewers also gained insight into the methods of solution used by the 
students and the kinds of misconceptions they have. This kind of 
information is extremely helpful in writing test questions with 
attractive distractors. In addition, the tapes provide a means for 
other ETS staff members to obtain knowledge of students' thought 
orocesses in problem solving. 
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APi'ENDIX A 

Item Interview Record 



I . 



II . 



What 

(A) 



major concepts and/or 

Solution of a simple 
involving a fraction. 



abilities are being tested? 
first degree equation in one 



unknown 



• Were the major concepts 
Yes □ No □ 

2. Was a different concept 
Technique of Solution 



and/or abilities tested? 
and/or ability tested? YesD 



NoD 



. ABODE 

(A) Option selected □ O 0 □ □ Right 0 Wrong □ 

<B> “ hat mcth ° d you use to answer this question? 

1. Notes on method used 



Omi t □ 



Procedures followed 

(a) steins options □ 

(b) options — * marked 

Reasons 



for students' 



(a) 

(b) 

(c) 

(d) 

(e) 

(f) 

(g) 

(h) 

(i) 

(j) 
00 
( 1 ) 
Cm) 



choice □ 
choice 



eliminated some options 

random guess 0 
faulty reasoning □ 
specific determiner □ 
lack of necessary knowledge □ 
misconception □ 
computational error □ 
ambiguity in the question □ 
too time-consuming □ 
correct answer not 
could not find his 

mistake in marking answer sheet 
other — specifyQ 



guessed □ 



recognized because 
solution among the 
□ 



of f o rm □ 
options □ 
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Test Interview Record 




I. Opinions about the items 

A. Were the content and abilities tested Yes □ 

appropriate to you? 

List of questions: 



B. Do you think all of the questions Yes □ 

were clearly stated? 

List of questions: 

► 

^ C. Do you think any of the questions Yes □ 

were tricky? 

List of questions: 



D. Do you think any of the questions took Yes □ 

too long t«o work? 

List of questions: 



E. Were there any questions you particularly Yes □ 
liked? 

List of questions: 



F. Were there any questions you particularly Yes □ 
disliked? 

List of questions: 



II . 



Opinions about the test 



A. 


Do you think the 
abou t r igh t □ ? 


test was 


too 


easy 


□ , too 


di f f i cul t 


B. 


Was enough time 


allowed to 


do 


the 


test? 


Yes □ 


C. 


Other comments : 















No □ 



No □ 



No □ 



No □ 



No □ 



No □ 



No □ 
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APPENDIX B 



Procedures Used In Carrying Out The Study 



Cooperation in conducting the study was solicited from four 
high schools within a 15-mile radius of Princeton: two suburban 

high schools, one private preparatory school, and an urban high 
school. Following are lines of communication used in the four 
schools: 

1. Principal to Mathematics Coordinator 

2. Curriculum Director to Mathematics Coordinator 

3. Headmaster to Mathematics Department Chairman to Teacher 

4. Principal to Vi c e -P r in c ipal to Mathematics Department 
Ch a i r man . 

The actual work in each school was done in close collaboration 
with the last person indicated in each of the four cases above. 

Since the first person contacted in each school needed to describe 
the study either to the school board or to the mathematics staff 
members, a brief description of the purpose and nature of the study 
was prepared and sent to the schools. A copy of the description can 
be found at the end of this appendix. 

A team of three ETS Mathematics Department members conducted the 
test administration and interviews in each school. ._Jhis number proved 
to be ideal since several different functions had to be performed. 

First, the test administration was facilitated by having several 
people available to give directions, distribute and collect; sup- 
plies, count test books, and proctor. Second, the availability of 
several persons expedited the rapid completion of tasks required to 
select the interviewees and arrange for the interviews. This selec- 
tion was done in collaboration with the mathematics teacher. While 
the answer sheets were being scored by one ETS staff member, a 
second arranged the interview room with appropriately placed chairs 
and recording equipment and attached school and student identifica- 
tion to the cassettes. Still another staff member helped to set up 
the interview schedule for the rest of the day after the interviewees 
were selected. This schedule was made to fit the student schedules. 

A system of notifying and paging students also had to be established, 
since the length of individual interviews varied somewhat. Finally, 
during the interviews one person did the actual questioning and 
introductory remarks on procedure, another recorded information in 
the notebook, and still a third ran the recorder and assisted in the 
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questioning. Although a team of three seemed to be ideal for this 
study as it was structured, it might be possible to restructure the 
interview techniques in such a way that two people could adequately 
complete a similar study. 

Generally, the schedule of activities during a day at a 
school was as follows: 



1. Enter school before classes begin and proceed to the main 
office. Explain purpose of visit again. 

2. Go to the mathematics department office. 

3. Review procedures again with the mathematics teacher. 

4. Go to classroom. 

5. Explain purpose of study to students. 

6. Administer test. 

7. Collect materials. 

8. Score answer sheets. 

9. Select three or four interviewees with advice of teacher; 
set up interview schedule. 

10. Ask permission of each interviewee to record interview. 

11. Interview each student for 1 to 1-1/2 hours. 
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Memorandum for : SCHOOLS PARTICIPATING 

IN A SMALL-SCALE ETS 
RESEARCH STUDY 



Subject: Brief description of an Date: May 7, 1970 

in-depth study of mathe- 
matics test questions From: Sheldon S. Myers 

Chancey 0 . Jones 



Up-to-now new test questions have been operationally pretested 
on random samples of appropriate groups. This pretesting then per- 
mitted the computation of normative information about the question 
in terms of per cent selecting the correct choice, the numbers 
selecting the other four choices, and measures of discrimination. 

This kind of information, while very useful for revising questions 
and developing final test forms, dealt entirely with group perfor- 
mance and did not tell us anything about what the questions measured, 
whether ambiguities existed, and what thought processes were 
stimulated by the questions. 

This study is an attempt to assess the feasibility of supple- 
menting the above approach with an in-depth investigation of What 
happens when individual students take a mathematics test. This 
approach is essentially clinical, in contrast to the normative 
approach of item analysis. 

The steps in this study are as follows: 

!• Administration (by Myers and Jones) in the 

morning of a 40-minute pretest of 25 questions 
to a class of 10-20 students who have had 3% 
years or more of college preparatory mathematics. 

2. Scoring of the answer sheets by Myers and Jones 
in the next half-hour. 

3. Interview in the rest of the day 4 to 5 
students from this class. The interviews 
will last, on the average, 60 to 90 minutes 
and will have the following purposes: 

a. Determine what method was used by the 
student in arriving at his correct or 
incorrect choices. 

b. Determine if the student misunderstood 
any part of a question. 

16 
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c. Determine if he was unduly delayed 
in any question and the reason. 

d. Determine if the student felt that 
any question was unfair. 

4. If the student agrees, the interview sessions 
will be taped. The identity of students and 
schools will be protected. 

5. The teacher will help us in advance to select 
the students to be interviewed. The inter- 
views will be held at times of greatest con- 
venience to the student 1 s schedule. 

Besides the purely professional objectives of the study in eval 
uating a technique and in gathering more information about questions 
the study is also consistent with the current trend of greater 
student involvement in processes that affect them. 



SSMzklb 
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APPENDIX C 
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Item Analysis for Total Group 



Ml 


No . 


Omi t 


A 


B 


C 


D 


E_ 


P + 


C 


i 


0 


0 


5 


70 


O 


0 


93.3 


E 


2 


6 


1 


3 


16 


2 


A 7 


62.7 


A 


3 


2 


63 


A 


2 


2 


2 


o 

00 


c 


A 


16 


5 


1 


25 


2 


26 


33.3 


B 


5 


0 


8 


26 


2 


10 


29 


3A . 7 


c 


6 


9 


5 


10 


A 2 


6 


3 


56.0 


D 


7 


12 


12 


5 


7 


37 


2 


A 9 . 3 


E 


8 


10 


6 


2 


0 


6 


51 


68.0 


C 


9 


18 


1 


6 


31 


5 


1A 


A 1 . 3 


E 


10 


30 


7 


5 


1 


6 


26 


3A . 7 


D 


11 


2 


A 


1 


3 


53 


12 


70.7 


B 


12 


7 


2 


23 


11 


9 


23 


30.7 


D 


13 


17 


1 


A 


5 


A3 


5 


57.3 


A 


1 A 


35 


A 


9 


A 


7 


16 


5 . 3 


C 


15 


15 


1A 


1A 


V 13 


A 


15 


17.3 


A 


16 


2 


AA 


9 


\ 5 


8 


7 


58.7 


B 


17 


21 


13 


15 


) 8 

1 


10 


8 


20.0 


E 


18 


30 


29 


1 


/ 0 


3 


12 


16.0 


A 


19 


21 


15 


6 


A 


20 


9 


20.0 


D 


20 


28 


2 


2 


6 


21 


16 


28.0 


B 


21 


21 


A0 


7 


0 


3 


3 


9 . 3 


E 


22' 


21 


25 


6 


7 


8 


8 


10.7 


A 


23 


A 9 


8 


6 


6 


A 


2 


10.6 


C 


2A 


8 


3 


31 


19 


A 


10 


25 . 3 


E 


25 


31 


11 


A 


7 


A 


18 


2A .0 




13 



-16- 



APPENDIX D 



Item A nalysis for Interview and Pretest: Croups 
and Interview Res u 1 1 s for Each Item 



Following are detailed statistical summaries and a description 
of the interview results for each question*. In the discussions that 
follow, a question is considered to be easy if approximately 80 per 
cent or more of the pretest population answered the question cor- 
rectly, of middle difficulty if approximately 60 per cent answered 
correctly, moderately difficult if approximately 30 per cent answered 
correctly, difficult if approximately 20 per cent answered correctly, 
and quite difficult if approximately 10 per cent or less answered 
correctly. 

Since the ability level of the population that took the 
original pretest differed somewhat from that of the population that 
took the revised pretest despite the random assignment of pretests 
to students, the observed statistical characteristics of the orig- 
inal version of an item and its revision cannot be directly compared. 
A comparison of the two groups with respect to four item statistics 
(n ...bers of students omitting the items, difficulty levels of the 
items, item-test correlations, and ability levels of students se- 
lecting the correct answer) is shown in the graphs in Appendix E. 
Because. of the lack of an obvious pattern for unrevised items in 
item-test correlation** (r-biserial) and ability level plots, these 
statistics will not be considered in the discussions concerning the 
results for revised items. There does, however, seem to be a 
definite pattern in the number of omits and difficulty level plots. 

In the following discussion, those revised items that are clearly 
outside of the pattern for unrevised items in at least one of these 
plots will be considered to have substantially different statistical 
characteristics in its original and revised forms. 



*Four of the questions that appeared in the pretest were from extant 
forms of the College Board Mathematics Level II Achievement Test and 
^ere used to relate new data to existing data. These four questions 
are not included in this report. 



**It should be noted that the criteria used in computing the 
r-biserials for the two pretests were somewhat different because of 
the revisions that had been made in some of the items. 
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Question 1. If = 5 , then x = 



(A) 75 < B > TO < c > 1 < D > 10 < E > 25 



Interview 


BASE N | 

1 

l 


OMIT 


A 


B 


c 

*15 


D 


E 


M TOTAL 

I 


a e scalc 


A E 


CRITERION 


Group 


ITEM NO. 1 

i 


Mq 


j«A 


M B 




m d 


m e 


fP TOTAL 


p + 


A 0 


r b.s 




1 
















100.0 








BASE N 


omit 


A 


B 


C 


0 


E 


M TOTAL 


A e SCALE 


a e 


CRITERION 


P r e t e s t 


235 


0 


4 


10 


19 7 * 


7 


17 


13 . 0 


BOARD 


12 . 3 


IS025 


Group I + 


ITEM NO. 


Mq 


m a 


m b 


M C 


m d 


m e 


p TOTAL 


p + 


A o 


r bis 




1 


0 . 0 


11.5 


10 . 0 


13.3 


10 . 6 


12.6 


1.00 


00 

o 


9 . 0 


0.26 




BASE N 


OMIT 


A 


B 


C 


0 


E 


M TOTAL 


a e scale 


a e 


CRITERION 


P v o ► q n t* 


235 


0 


3 


24 


188* 


8 


12 


13.0 


BOARD 


12 . 8 


IS025 


r reces L 
Group II 


ITEM NO. 


Mo 


m a 


m b 


M C 


Mp 


m e 


P TOTAL 


p + 


A 0 


r bis 




1 


0.0 


12.0 


11.0 


13.6 


9 . 3 


10 . 5 


1.00 


0 . 80 


9 . 6 


0.43 



This question was included in the test in order to determine 
the student’s ability to solve a simple linear equation involving 
fractions. It appeared as the first item because it had been judged 
to be easy. All of the interviewees answered the question correctly; 
twelve of the interviewees multiplied both members of the equation by 

0.2, (whereas the remaining three replaced 0.2) with .1, obtaining 

5 

5x = 5, and then divided both members by 5. Some of the interviewees 
were surprised to find such an easy question on the test but did not 
object to it. 

The data obtained from regular pretesting indicated that the 
question was easy since a majority of the students answered the 
question correctly. The topic is evidently quite familiar to the 
pretest population. 



^Pretest Group I is the population that took the original version of 
the test. 



^Pretest Group II is the population that took the revised version of 
the test. 
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Question 2. If the following instructions to a computer are carried out 
in the order specified, what value of S should be written in 
instruction 6 ? 




1 . 

2 . 

3. 

4. 

5. 



6 . 



Let S = 0. 

Let X = 5. 

Let the new value of S equal the old value of S plus 
the value of X. 

Increase X by 2. 

If X <8, go back to instruction 3. Otherwise, go on to 
instruction 6. 

Write the final value of S, 



(A) 0 (B) 5 (C) 7 (D) 9 



(E) 12 
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Question 2. If the following instructions to a computer are carried 
out in the order specified, what value of S should be 
written in instruction 6 ? 

1. Let S = 0. 

2. Let X = 5. 

3. Let the new value of S equal the previous value 
of S plus the value of X. 

4. Increase X by 2. 

5. If X < 8, go back to instruction 3. Otherwise, 
go on to instruction 6. 

6. Write the final value of S. 



(A) 0 (B) 5 (C) 7 (D) 9 (12) 12 
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For some time the committee has been interested in finding out 
the extent to which candidates have been exposed to computer instruc 
tion in order to determine the appropriateness of including computer 
questions on the examination. This question was deliberately 
included in the pretest to obtain that kind of information. Of the 
fifteen students interviewed, seven students had been exposed to 
some computer instruction and solved the problem on that basis with 
five of them obtaining the correct answer. Of the eight who had 
had no exposure to the computer, one student omitted the question 
and seven treated it as a task in logical reasoning with six of 
these solving the problem correctly. It is obvious that although 
a computer background might be helpful it is not necessary for 
solving this problem. Consequently the results of this question 
did not give the committee all of the information it had hoped to 
obtain. 

Since eleven of the fifteen interviewees selected the correct 
answer, a question of this kind that can be answered with or with- 
out knowledge of the computer seems to be an appropriate question 
to include on the test. 

Several interviewees were confused by the use of the word "old 
They were not sure whether this referred to the value of S that was 
initially stated or to the value that they had just obtained. As 
a result of their comments, the word "old" was replaced by the 
word "previous , 11 and both versions of the question were pretested 
on the usual pretest population. 

The results of the pretesting indicated no appreciable 
difference in the statistical characteristics of these two versions 
However, the infrequency with which the item was omitted by the 
regular pretest population supports the surmise based on the inter- 
views that the question is an appropriate one for the intended 
Dopulat ion . 
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Que s t ion 



4. 



cos 50° 
sin 40° 



(A) -1 



(B) 0 (G) 1 



(D) tan 10° (E) cot 10° 
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The committee deliberately included this question in order to 
see whether students would recognize the equality of cofunctions of 
complementary angles. Of the fifteen interviewees, eleven answered 
the question correctly. Of these, four answered on the basis of 
cofunctions of complementary angles, five used the identity 



sin|^ - x^ = cos x without thinking about complementary angles 

, cos 50° cos (45° + 5°) j , c ^ \ 

used ; r-*A“T = — ; rrr o and used the sum formulas, c id 

sin 40 sin (45 -5) 



, one 
one 



sketched a right triangle and used the definition of trigonometric 
functions. The four who answered the question incorrectly thought 

that — — T-Trer is equal to cot (50° - 40°). 
s in 4 U 



For the regular pretest population the question was moderately 
difficult. ''cot 10°, 11 the moot popular distracter was selected by 
about the same number of students as answered the question correctly. 
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Question 5. The set of all real x such that v/x 2 = -x consists of 

(A) zero only 

(B) nonpositive real numbers only 

(C) positive real numbers only 

(D) all real numbers 

(E) no real numbers 
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The committee included this question on the test to determine 

whether students would associate /x 2 with |x| and whether they were 

familiar with the convention that / denotes the non-negative root. 
Among the interviewees, eight answered correctly; three of these used 

the fact that / denotes the non— negative square root, two used the 
concept of absolute value, two selected the correct answer by 
examination of the available options, and one substituted numbers 
such as -1, 0, 1, and chose the correct answer. 



Two of the seven interviewees who answered the question incor- 
rectly thought that -x was a negative number and since / is posi- 
tive chose "no real numbers,” two thought J could be either posi- 
tive or negative and selected "all real numbers," and the remaining 
three selected "zero" for different reasons: one of the latter had a 

very definite misconception that |x| = ~x only if x = 0; one believed 

V^2" being positive and -x being negative implied x = 0; one substituted 
some numbers . 



This was a moderately difficult question for the regular pretest 
population and the pattern of responses was somewhat similar to that 
of the interview group but with "no real numbers" being selected 
by more candidates than the correct answer was. 

Although this question is probably appropriate, it appears that 
it involves two concepts about which many students are confused. 
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Oues tion 



6. For all xand y such that xy * 0, let f(x, y) = 
Then f(x, -x) = 



xy 

x 2 + y 2 



(A) -x 2 <B) (C) (D) 0 (E)i 
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The committee included this question to determine whether students 
could solve a simple substitution problem involving a function of 
two variables. Among the interviewees, twelve answered the question 
correctly and none of them were confused by the function notation; 
all twelve substituted -x for y and simplified. Two other inter- 
viewees also using this method made errors and one student was 
confused by the notation and omitted the question. In talking with 
the students, it appeared that little or no attention was paid to 
the qualifier "xy ^ 0." In general the interviewees did not con- 
sider this question to be difficult#. 



On the regular pretest population this question was of about 
middle difficulty. The relatively small number of omits seems to 
support the indication obtained from the interviewees that the 
notation was not confusing. 
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Ques tion 8. If x 4 - 1 = 80, then x 3 + x 2 +x + 1 could equal 

(A) -81 (B) -80 (C) -60 (D) -40 (E) -20 
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This question was included by the committee in order to determine 
whether the students would use x 4 -l =. (x-1) (x 3 +x 2 + x+l) in determining 
a value for the cubic expression. None of the interviewees used 
this factorization of the quartic in solving the problem. All of the 
interview group who attempted the question solved the equation 
x^^Sl and substituted. Of the thirteen who answered the question 
correctly, five immediately substituted -3 since all options were 
negative, one Substituted -3 since x 3 was the dominating term, and 
seven others substituted 3 first and upon obtaining a positive value 
substituted -3. The one student who answered the question incorrectly 
substituted 3 and obtained 40 and assumed that substituting -3 would 
yield -40. One interviewee omitted the question. The question was 
not difficult for the interview group although they did not use the 
intended method. 

The question was about middle difficulty for the pretest 
population and the number of omits was small, indicating that the 
topic was familiar to the pretest group; however, -40 was the most 
attractive distracter. 
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Question 10. A polynomial P(x) with real coefficients has three zeros. 

If two of the zeros are Oand i, then P(x) could be 

(A) x 2 + 1 (B) x 3 - 1 (C) x 3 + 1 

(D) x 3 - x (E) x 3 +x 
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Pretes t 
G r o u n II 



Question 10. A polynomial equation, P(x) = 0, with real coeffi- 
cients has three roots. If two of the roots are 0 
and i, then P(x) could be 

(A) x 2 4* 1 (B) x 3 - 1 (C) x 3 4* 1 

(D) x 3 - x (E) x 3 4* x 
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The committee Included this question to determine the 
familiarity of the terminology "zeros of a polynomial.” The question 
was answered correctly bv nine of the interviewees; six of these 
were familiar with the terminology and three were not familiar 
with it but correctly interpreted it to mean the roots of a poly- 
nomial equation. The remaining six did not understand the question; 
five of them omitted it and one guessed incorrectly. Six of the 
interviewees understood the concept of conjugate pairs of complex 
roots of polynomial equations with real coefficients and used this 
concept to solve the problem with two of them using the factor 
theorem and the other four eliminating options by substituting 
roots. The remaining three who answered the question correctly 
did so by substituting the given zeros in the options. 



Since nine of the interviewees were not familiar with the 
terminology, it appeared that the question might not be appropriate 
for the intended population. The item was revised by changing 
'zeros of a polynomial P(x)’* to ''roots of a polynomial equation 
P (x) = 0 . ! * 



On the regular pretest population the difficulty and the number 
of omits decreased substantially from the original version to the 
revised version. Although the revised version of the item was 
moderately difficult, it seem to be more understandable to the 
population than was the original form. 

' •* 
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Questlon 11 . sin 2 20 + cos 2 20 = 



(A) \ ( B ) ^ (C) Y (D) 1 (E) 2 
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The purpose of this question was to test the specific trigono- 
metric identity, sin 2 x + cos 2 x = 1. All of the interview group 
answered the question correctly. Most of the students were not con- 
fused by the coefficient of 0 although one student thought that the 
intent of 20 was to confuse students and one thought it was "tricky" 
in that 0 was not defined. 

For the pretest population the item was of about middle diffi- 
culty. The most popular distracter was 2 which could be obtained 
by incorrectly assuming that sin 2 20 4- cos^20 - 2(sin^0 + cos 2 0) . A 
relatively small percentage of the students in the pretest population 
omitted this question. 
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Que s tion 12. If n is a positive integer, then n! is divisible by 9 if and 
only if 



(A) n 23 (B) n g 6 (C) n S 9 

(D) n is a multiple of 3 (E) n is a multiple of 9 
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The committee included this question in order to test the 
meaning of n! and the terminology "if and only if." The question 
was answered correctly by eight of the interviewees, incorrectly 
by six, and omitted by one. All but one of the students knew the 
meaning of n! but only one of the interviewees thought about "if and 
only if ' 1 in ruling out options. Most of the others who answered 
the question correctly felt that they selected the answer that was 
best" in some sense which they could not always express or was 
most "complete" or "inclusive." The best student in the interview 
group answered the problem correctly without thinking about "if 
and only if." 

It was a difficult question for the regular pretest population; 
however, very few students omitted the question. Almost three 
times as many students selected distracter (E) as selected the correct 
answer. Choice (E) would have been a correct answer for "if" 
instead of "if and only if." The data seem to support the imnression 
obtained from the interview group that "if and only if" is not 
generally understood by students at this level. 
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Ques t ion 



13. For x > 1, the graph of y = l°g x x intersects the line 
x = ir at y = 
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The objective of this question was to determine whether 
students recognize that log x x = 1 for x>l and that y = log x inter- 
sects x = n at y = 1 . Among the interviewed students, nine answered 
the question correctly, five answered the question incorrectly, and 
one omitted the item. It appeared from the interviews that the 
students either knew how to cope with the problem or guessed. The 
question was of middle difficulty for the interview group. 



The question was slightly above middle difficulty for the pre- 
test population with about one-fourth of this group omitting the 
q ue s tion . 



Question 14. If the matrix A = [ 1 0 -1] and the matrix B = 



, then 



the matrix AB = 
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Question 14. If matrix A = ^ and matrix B = q 

then matrix AB = 
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This question was included by the committee in order to determine 
whether or not the topic of matrices is appropriate for this popula- 
tion. Only two of the interviewees had studied matrices in their 
course work with one of these answering the question correctly. 

Reports had been written on the subject by two other students and 
both answered the question correctly. Of the remaining eleven who 
were not familiar with the topic, three guessed incorrectly and 
eight omitted the question. This question was clearly inaupropriate 
for this group; however, since some of the students indicated that 
2 by 2 matrices might be more familiar to students, a new question 
was written to test multiplying 2 by 2 matrices rather than the 
multiplication of a 1 bv 3 matrix and a 3 by 1 matrix. 
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Both of the questions were quite difficult for the pretest 
population probably because many students had not been exposed to 
the topic. The decrease in the number of omits from the original 
item to the new one on the regular pretest population substan- 
tiated the indication from the interviewees that students might be 
more familiar with 2 by 2 matrices than with nonsquare matrices; 
however, both questions were extremely difficult and were about 
the same level of difficulty. Although only a small nercentage of 
the students omitted the question, the pattern of responses seems 
to indicate that many students were not familiar with matrix 
multiplication but attempted to devise a reasonable definition of 
mul tipli cation . 



Question 16. Which of the following is the set of all real numbers x 
such that jx j - 5 = |x - 5| ? 

(A) The set of all real numbers 

(B) {5} 

(C) {-5} 

(D) {x: x ^ 0} 

(E) (x:xi 0} 
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The purpose of this question was to determine whether the students 
could find the solution of an inequality involving absolute value. 

The question was answered correctly by eight of the interviewees and 
incorrectly by seven. Those answering the question correctly did so 
by substituting negative, positive, and zero values into the expres- 
sion. All of the others attempting to find the solution also used 
substitution but did not consider positive, negative, and zero values. 
The question was slightly above middle difficulty for this group. 

For the pretest population, the question was also slightly above 
middle difficulty, and there were yery few omits. 
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Question 17. If two different numbers are chosen at random from the 
set {1, 2 , . . 6}, what is the probability that their 

sum is greater than 6 ? 
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The objective of this question was to measure the students 1 
understanding of simple probability. The question was answered 
correctly by six of the interviewees, but one had guessed and one 
had gotten the right answer for the wrong reason. The item was 
omitted by six, answered incorrectly by three. Six of the students 
stated that they did not think that they had sufficient background 
for solving this problem, and at least two other students misread 
the question. The question was above middle difficulty for the 
interview group . 

The question also Droved to be moderately difficult for the 
regular pretest population although the number of omits was not 
excessive. 
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Ouestion 18. Let f and g be functions such that g(f(x)) = x for all x. 

If (5, 0) is a point on the graph of y = g(x), which of 
the following equations could define f ? 

I. f(x) = x - 5 

II. f(x) = 3x +5 

III. f ( x) = x 3 +5 



(A) I only (B) II only (C) III only 
( D) 1 and II only ( E) 11 and III only 
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Question 18. If f and g are inverse functions and if (5,0) is a 
point on the graph of y = g(x), which of the follow- 
ing could define f ? 



I. f(x) = x - 5 

II. f(x) = 3x + 5 

III. f(x) = x 3 + 5 
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Thic. question was included to test the concept of inverse 
function by the definition g(f(x)) = x without stating that f and g 
are inverse functions. Four of the interviewees recognized that f 
and g are inverse functions, but one of these did not realize that 
the point (5,0) on the graph of g implies that the point (0,5) is on 
the graph of f. Of the remaining eleven students who did not recognize 
that f and g were inverse functions, three attempted to answer the 
question but selected an incorrect answer, whereas the other eight 
omitted the question. Some of the interview group did not like this 
item type which consists essentially of three true-false questions, 
all of which must be answered correctly to receive credit. 
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It appears that the testing of inverse functions using defini- 
tion instead of terminology was not appropriate for the interview 
group. However, the concept of inverse functions does appear to 
be appropriate for students at this level. Since many of the 
students indicated that they were familiar with the concept of in- 
verse functions but had not recognized its formal definition, the 
item was revised as shown above. 

Both of the questions were considerably above middle difficulty 
for the regular pretest population; however, the item type may have 
contributed to some extent to the difficulty. Although the number 
of omits was approximately the same on both versions, the difficulty 
level dropped substantially from the original to the revised 
version. 

This decrease in difficulty tends to substantiate the indi- 
cation given by the interviewees that the concept of inverse 
functions is more familiar to students than is the ftfrmal defini- 
tion. 
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Figure 1 



Question 19. In Figure 1, it the equation of the graph in I is y = f(x) # 
then the equation of the graph in H could be 

(A) y = 2f(x + 7) - 5 

(B) y « 2f(x - 7) +5 
(G) y « 2f (x •• 7) +5 

(D) y = 2f(x •• 7) -5 

(E) y = 2f ( — x) +5 
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The purpose of this question was to determine whether students 
could recognize the effect that translating a graph would have on 
the equation of the original graph. The question was answered 
correctly by six interviewees, incorrectly by two, and omitted by 
seven. The students answering Correctly either used concepts of 
translation, substitution of points, or a combination of both. The 
item was somewhat above middle difficulty for this group. It seemed 
apparent in the interviews that some of the students were confused 
about the correspondence between the direction of shift involved 
in a translation and the sign of the number to be added to the 
abscissa or the ordinate. 

The question was difficult for the pretest population. The 
fact that option D was selected by more students than was the correct 
answer tends to support the impression that a number of students are 
confused about the "sign" associated with a translation. 
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Question 20. Which of the following functions satisfy |f(x) | g 1 for all 
but finitely many numbers in the interval 0 < x § 2ir ? 

I. f(x) = sec x 

II. f(x) = esc x 

III. f(x) = tan x 



(A) I only (B) II only (C) III only 
(D) I and II only (E) I. II. and III 
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Question 20. For which of the following functions is |f(x)|Sl 
for all x in the interval 0 < x § 2jt for which f 
is defined? 



I. f(x) = sec x 

II. f(x) = esc x 

III. f(x) = tan x 
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This question was included to test the range of trigonometric 
functions and the meaning of absolute value. The ranges of the 
given functions were considered by nine of the interviewees; six of 
these answered the question correctly and three incorrectly. Four 
considered the reciprocals of sec and esc and their ranges and 
drew conclusions about the ranges of sec and esc. All four of 
these students answered the question correctly; however, one of 
these incorrectly eliminated the tangent because it was not defined 
on the entire interval. The question was omitted by one and 
guessed incorrectly by one. 

Almost without exception the students did not understand the 
phrase "all but finitely many 11 and tended to ignore it in solving 
the problem. 



th e 



The topic was evidently a familiar one; however, because of 
confusion regarding the wording the question was revised. 



Both versions of the question were moderately difficult for the 
pretest population. However, again the item type may have contri- 
buted to some extent to the difficulty of the question. Revising 
the item did not appear to make any appreciable difference in the 
statistical characteristics of the item. 
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This question was included in order to test the concept of limit 
and technique for determining a limit when both the numerator and 
denominator of a fraction tend to 0. Of the fifteen students inter- 
viewed only four used a correct technique; one correctly answered the 
question using L'Hospital's rule, and three rationalized the 
numerator with two of them obtaining the correct answer. Of the 
remaining eleven students, four substituted 0 in the numerator and, 
since it was 0, disregarded the denominator and therefore answered 
the qu estion incorrectly; three others estimated the value of 
2 -• /A -- 1 for values of t close to 0 and guessed, with one guessing 
t 

the correct answer. One additional student made a random incorrect 
guess and three students omitted the question. Most of the students 
seemed to have been exposed to the concept of limit but either did 
not remember or had not studied the technique required to solve 
this problem. 

Since most students are familiar with rationalizing denominators 
rather than numerators the item was revised so that the limit could 
be obtained by rationalizing the denominator. 

For the regular pretest population, both the number of omits 
and the difficulty level increased slightly from the original to 
the revised form. Both versions of this item were quite difficult. 
The concept of limit may be an appropriate topic for this group, 
but perhaps it should be tested with a less sophisticated technique. 
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Ques tion 



22. In a complex plane, the set of points z such that | z | = 1 
is a region whose boundary is 
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(A) two parallel lines 

(B) an equilateral triangle 
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Question 22. Ifa complex number, a + bi, is plotted in a 

coordinate plane as the point (a, b), then the set 
of points such that |a + bi| 2 1 is a region whose 
boundary is 
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The intent of this question was to determine the extent to which 
the complex plane and the absolute value of a complex number are 
understood by students at this level. Only two of the interviewees 
understood the concepts involved in this question. One of these 
replaced z by /x z + y z , and the other interpreted z as an ordered 
pair and z as distance. Most of the others trying the question 
guessed and four students omitted the question. It appears that 
these concepts were not appropriate for the interview group. 



So few students understood what is meant by "complex plane” 
that the question was revised. 



Both versions of this question were quite difficult for the 
pretest population and no appreciable differences were evident in 
the statistical characteristics. It appears from the pattern of 
responses that a number of students may have randomly guessed. 
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Question 23. Which of the following is the center of the conic section 
whose equation is 4x 2 - 9y 2 + 16x +90y - 245 = 0 ? 

(A) (-2,5) (B) (2, -5) (0(2,5) 
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Oues t ion 23. Which of the following is the center of an ellipse 
whose equation is 

4x 2 + 9y 2 + I6x - 90y - 245=0 ? 

(A) (-2,5) (B) (2, -5) (C) (2, 5) 

(D) (5, -2) (E) (5, 2) 
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The intent of this question was to test the students' ability 
to find the center of a conic section. Five of the fifteen inter- 
viewees used the correct method of completing the square; four 
obtained the correct answer, and one selected the coordinates with 
the wrong signs. Of the remaining ten, two guessed incorrectly 
and eight omitted the question. Several of the interviewees were 
not familiar with finding the center of a conic section although 
they stated during the interview sessions that they had learned 
how to find the center of a circle in a coordinate plane. 

Because o f the interviewees' un f ami lia r i ty with "conic section," 
this phrase was replaced with "ellipse." For the regular pretest pop- 
ulation both versions were difficult. There was a noticeable decrease 
in the number of omits from the original to the revised version; 
however, there was no appreciable change in the level of difficulty. 
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Question 24. How many real numbers are solutions of the equation 
x 2 = 2 X ? 



(A) None (B) One (G) Three 

(D) Four (E) Infinitely many 
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Question 24. Use the graphs of y = x 2 and y=2 x to determine 
the number of real solutions of the equation x 2 = 2 X . 

(A) None (B) One (C) Three 

(D) Four (E) Infinitely many 
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The intent of this question was to determine the students 1 
ability to find the number of solutions to an equation using a 
graphical method. Only three students in the interview group 
obtained the correct answer. Only one of these used a graphical 
method; the other two found two solutions by inspection, and one 
of these guessed there should be a negative solution and the other 
incorrectly thought 8 was a solution. The other students in the 
interview group either found one solution by inspection and selected 
,, l n for the answer or found two solutions by inspection and guessed 
" infinitely many . 11 

It seems apparent that students are not familiar with using 
graphs to determine the number of solutions of an equation of this 
type. In order to determine whether the students could find the 
number of real solutions if they were told to use graphs, the 
question was revised as shown above. 




For the regular pretest population, there was an increase 
in the number of omits from the original to the revised version; 
however, there was no appreciable difference in the difficulty 
levels. Both versions of the item are quite difficult. 
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Question 25. Which of the following lines are asymptotes of the graph of 




(A) x = 3 only 

(B) y = 3 only 

(C) x = 1 and y = 3 

( D) x = 3 and y = 3 

(E) x = 3 and y * 1 
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The intent of this question was to determine whether students 
could find the vertical and horizontal asymptotes of the graph of 
an algebraic equation. Seven of the interviewees answered the 
question correctly, three answered incorrectly, and five omitted the 
item. One of the students answering the question correctly saw that 
the equation was undefined for x-3 , solved for x in terms of y, and 
found that for y-1 the equation was also undefined. Two other 
students recognized the vertical asymptote and found the horizon- 
tal asymptote by finding the number that y approached as x became 
very large. The other four students also recognized the vertical 
asymptote, substituted values for x, and concluded that y got close 
to 1 but did not become 1. Three students found the vertical asymp- 
tote only and selected option A. The question was moderately diffi- 
cult for the interview group. 

The question was also moderately difficult for the pretest 
population. Option A was the most attractive distracter and was 
selected by more students than was the correct answer. 
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Difficulty - Original Pretest 



Fig. 1. Difficulties of Corresponding Items in Original 
Pretest and Revised Pretest 
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Fig. 2 . N umbers of Students Omitting Corresponding Items 
in Original Pretest and Revised Pretest 
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Fig. 4. Mean Ability Scores of Students Answering Corresponding 
Items Correctly on Original Pretest and Revised Pretest 
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